Evaluating Content-Related Validity Evidence Using a Text-Based Machine Learning Procedure

Autor/inn/en	Anderson, Daniel; Rowley, Brock; Stegenga, Sondra; Irvin, P. Shawn; Rosenberg, Joshua M.
Titel	Evaluating Content-Related Validity Evidence Using a Text-Based Machine Learning Procedure
Quelle	In: Educational Measurement: Issues and Practice, 39 (2020) 4, S.53-64 (12 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Anderson, Daniel) ORCID (Rowley, Brock) ORCID (Stegenga, Sondra) ORCID (Irvin, P. Shawn) ORCID (Rosenberg, Joshua M.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0731-1745
DOI	10.1111/emip.12314
Schlagwörter	Content Validity; Artificial Intelligence; Test Items; Vocabulary; Models; Academic Standards; Science Education; Alternative Assessment; Evidence; Oregon + Suchen Sie Ihr Suchwort? Künstliche Intelligenz; Test content; Testaufgabe; Wortschatz; Analogiemodell; Naturwissenschaftliche Bildung; Evidenz
Abstract	Validity evidence based on test content is critical to meaningful interpretation of test scores. Within high-stakes testing and accountability frameworks, content-related validity evidence is typically gathered via alignment studies, with panels of experts providing qualitative judgments on the degree to which test items align with the representative content standards. Various summary statistics are then calculated (e.g., categorical concurrence, balance of representation) to aid in decision-making. In this paper, we propose an alternative approach for gathering content-related validity evidence that capitalizes on the overlap in vocabulary used in test items and the corresponding content standards, which we define as "textual congruence." We use a text-based, machine learning model, specifically topic modeling, to identify clusters of related content within the standards. This model then serves as the basis from which items are evaluated. We illustrate our method by building a model from the "Next Generation Science Standards," with textual congruence evaluated against items within the Oregon statewide alternate assessment. We discuss the utility of this approach as a source of triangulating and diagnostic information and show how visualizations can be used to evaluate the overall coverage of the content standards across the test items. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01